System and method for image-based surface detail transfer

ABSTRACT

A system and method, called Image-Based Surface Detail Transfer, to transfer geometric details from one surface of an object in an image to another with simple 2D image operations. The basic observation is that, without knowing its 3D geometry, geometric details (local deformations) can be extracted from a single image of an object in a way independent of its surface reflectance, and furthermore, these geometric details can be transferred to modify the appearance of other objects directly in images. Examples are shown including surface detail transfer between real objects, as well as between real and synthesized objects.

BACKGROUND

1. Technical Field

This invention is directed toward a system and process, calledImage-Based Surface Detail Transfer (IBSDT), for transferring geometricdetails from one surface in an image to another surface in another imagewith simple 2D image operations.

2. Background Art

Changing the appearance of an object by adding geometric details isdesirable in many real world applications. For example, one may want toknow what a wall might look like after adding some geometrical bumps onthe wall, or one may want to know what a person might look like afteradding/reducing wrinkles on his/her face, and so on. Adding geometricdetails to an object typically requires modeling both the object and thesurface details. It is usually not trivial to build a 3D model for areal object. It is also typically tedious and labor intensive to modeland create surface details with existing geometric modeling tools. Bumpmapping [3] has been used as an alternative to adding geometricaldetails to an otherwise smooth object. But constructing visuallyinteresting bump maps requires practice and artistic skills.

Computer vision techniques have been very helpful for modeling realworld objects as well as the surface details. These techniques includelaser scanner, steror algorithms, shape from lighting variation [8, 17],and shape from shading [10,9], among others. There are, however, manydifficulties in the techniques used to model these real world images.Some of these techniques require specialized equipment. Many othertechniques require at least two images for each object to be modeled,and it may be difficult to capture the high resolution geometricaldetails required for photo-realistic modeling robustly. Although shapefrom shading technique only requires a single image, this method usuallyrequires detailed knowledge of the lighting condition and reflectancefunctions.

One use for changing the appearance of an object by adding geometricdetails is in the context of aging simulation. Aging simulation of humanfaces has applications in computer games, entertainment, cosmetics andvirtual reality. Skin aging is a complex process that depends onmultiple factors such as age, race, gender, health and even lifestyle.Anatomically, skin is attached to the underlying muscle by connectivetissues and the attached end of a muscle is fixed to the skull. Facialappearance changes as the consequence of the gradual aging change of allof the facial components and the comprehensive interactions among thesecomponents. In spite of the difficulty of the problem, varioustechniques have been developed to analyze and synthesize facial agingeffects. These methods can be roughly classified into three categories:model-based, image-based, and learning-based.

The model-based approach for facial aging effects is closely related toprevious work on skin deformation simulation and skin texture synthesis.Wu et al. [18] proposed a three-layered Elastic Membrane Model forfacial wrinkle simulation where “the deformation of skin is activated bythe simulated muscle layer, constrained by the connective tissue layerand decided by a biomechanical model”. The skin model is computed withthe aid of the feature points selected on the reconstructed face model.The wrinkles generated from the skin model are composed with real faceimages to produce the image of an aged face. An improved model wasreported by Boissieux et al. [2], where the thickness and the mechanicalproperties of each skin layer are considered. This approach providesgood insight into the nature of the aging process and can be used asguidelines in cosmetic and medical applications. In general, thisapproach requires 3D geometry information to perform physicalsimulation. The results are usually not as photorealistic as the imagebased approaches.

Boissieux et al. [2] developed an image-based method that uses eightgeneric masks generated from real photos of the aged people. Each maskcontains quantitative information about the amount, shape and intensityof wrinkles with respect to gender, facial shape, and expression. Tocustomize the face of a particular person, the wrinkle intensity (ordepth) is computed and the mask is warped onto that face. Thecomposition of the warped mask image and the image of the specific faceforms the texture map of the final 3D model of the face. Because thegeneric masks contain mainly the wrinkle information, othermorphological changes on the face due to aging cannot be reflected. Anadditional limitation of this method is that it cannot make an old facelook younger. An additional-image based method was reported by Bursonand Nancy [4]. It computes the differences of the aligned images of ayoung face and an old face. Given the image of another young face to beaged, the difference image is warped and added to this face to make itlook older.

Learning-based approaches try to establish a statistical model for theaging process without understanding the underlying mechanisms. Lanitiset al. [11] suggested a linear face model of 15 parameters, obtained byPrincipal Component Analysis (PCA) on a set of normalized trainingexamples. By using the same set of training data after sorting itaccording to age, they are also able to find a so-called aging functionthat relates the model parameters to the age. Choi [5] uses a PCA methodto find the age related components for both skull and skin changes. Bycarefully choosing and normalizing the training examples, he is able tosimulate the aging effect with the first principal components from bothskull and skin data. The learning based approach is powerful because itdoes not rely on detail domain specific knowledge. It does, however,require a careful selection of the training data used.

It is noted that in the preceding paragraphs, as well as in theremainder of this specification, the description refers to variousindividual publications identified by a numeric designator containedwithin a pair of brackets. For example, such a reference may beidentified by reciting, “reference [1]” or simply “[1]”. A listing ofthe publications corresponding to each designator can be found at theend of the Detailed Description section.

SUMMARY

The present invention is directed toward a system and process thatovercomes the aforementioned limitations in systems for transferringgeometric details from one surface to another by using simple 2Doperations and without knowing the actual geometric information of thesurfaces. In particular, the invention uses a novel technique, calledImage-Based Surface Detail Transfer (IBSDT), to capture the geometricaldetails of an object from a single image in a way that is independent ofits reflectance property. The captured geometrical details can then betransferred to the surface of a similar object to produce the appearanceof the new surface with added geometrical details while its reflectanceproperty is preserved.

The computer-implemented system and process for transferring geometricaldetails from an object in a first image to a similarly shaped object ina second image operates as follows. A first image depicting an objectwhose surface details are to be transferred and a second image depictingan object of similar shape as depicted in the first image are input intothe system. The two images ideally should be taken under the same, or atleast similar, lighting conditions. For images taken under completelydifferent lighting conditions, one of several known relightingtechniques may be used to compensate for the lighting difference betweenthe images. For instance, the color ratio between the images taken underdifferent lighting conditions can be used to modify at least one of theimages such that the lighting between the images is the same. Similarly,the color difference (instead of ratio) between image pairs can be usedto modify one of the original images so that it matches the lightingconditions of the other.

The two objects in the two images are then aligned. If the objects areof a simple geometrical shape, such as a sphere, square or rectangle,alignment is typically performed by simple rotation, translation andscaling. If the objects are more complex, such as that of a human face,the objects in the images are usually aligned via image warping. Thisinvolves putting markers on the feature points of the objects in theimages. The feature points of the objects are then aligned by warpingthe coordinates of the feature points in the first image to thecoordinates of the feature points in the second image, thereby allowingthe corresponding pixel locations in the first and second images as wellas their respective intensities to be identified. One of many knownimage warping techniques may be used. In one embodiment the images arewarped by using Delaunay triangulation, a popular image warpingtechnique.

Once the images are warped, smoothed versions of the input images afterwarping are computed. This smoothing is preferably performed by applyinga Gaussian filter, but other known image smoothing techniques can alsobe used.

The ratio of the original intensity to the smoothed intensity is thencomputed for each pixel in the first input image after warping.

A new image, having the geometric details of the first input image, butthe material properties of the second, is then created by multiplyingthe ratio of the original intensity to the smoothed intensity of thefirst warped image by the smoothed intensity in the second image foreach corresponding pixel (correspondence as determined by the warpingoperation).

One embodiment of the invention simulates the aging or making youngerthe facial characteristics of a person. Two images are input, a firstimage depicting a face whose facial characteristics are to betransferred and a second image depicting a face that is to receive thefacial characteristics of the face in the first image. If the two imageswere taken under different lighting conditions, this can be compensatedfor by one of the various conventional relighting techniques. The facesin the first and second images are then aligned. A smoothed version ofthe faces in the first and second images are then computed. Thissmoothing is conducted either by applying a Gaussian filter or in someother way of downsizing or averaging the pixel intensity of the images.For each pixel in the face of the first warped image, the ratio of theoriginal pixel intensity to the smoothed intensity is computed. A newfacial image is then created by multiplying the ratio of the originalintensity to the smoothed intensity of the face in the first warpedimage by the smoothed intensity for each corresponding pixel in thesecond image. Thus, a person can be aged if the facial characteristicsof the face to be transferred depict an older face than the facialcharacteristics of the face in the second image. Likewise, if the facialcharacteristics of the face to be transferred depicts a younger face, aperson can be depicted as younger.

One issue with respect to the IBSDT technique is that it assumes thatthe surface reflectances are smooth. For objects with abrupt reflectancechanges such as small color spots, the IBSDT may confuse these colorspots with geometrical details. For example, the IBSDT technique is onlyintended to apply to skin, not eyebrows, eyeballs, or lips. Therefore,in one embodiment of the invention dealing with facial changes, theseregions are masked out so that they are not changed. Similar masking canbe used in other embodiments. As an alternative to masking, it may bepossible to separate these color variations from geometry variationsperhaps through learning or some other approach.

The IBSDT technique is particularly useful for adding geometric detailsto a real world object for which only a single image is available as itrequires only a single image for each object from which surface detailsare to be transferred. It also provides a simple way to capturegeometrical details of a real world object and apply it to othersynthetic or real world objects. Other advantages of this method arethat it is simple to implement and reliable. In a facial aging context,this invention allows a user to easily generate various aging effects aslong as they can obtain different people's facial images. The desiredfacial effects can be transferred from facial images of people ofdifferent sexes, races and ages. Additionally, the aging simulation isbi-directional, meaning that it can also make an old person lookyounger.

DESCRIPTION OF THE DRAWINGS

The file of this patent contains at least one drawing executed in color.Copies of this patent with color drawing(s) will be provided by the U.S.Patent and Trademark Office upon request and payment of the necessaryfee.

The specific features, aspects, and advantages of the present inventionwill become better understood with regard to the following description,appended claims, and accompanying drawings where:

FIG. 1 is a diagram depicting a general purpose computing deviceconstituting an exemplary system for implementing the invention.

FIG. 2A is a flow diagram depicting the general process of the systemand method according to the present invention.

FIG. 2B is a flow diagram detailing the image alignment process actionof FIG. 2A.

FIG. 3 is a series of images illustrating the results of transferringthe geometrical details of a synthetic sphere to a nectarine.

FIG. 4 is a series of images illustrating the results of transferringthe geometrical details of a real orange to a nectarine.

FIG. 5 is a series of images illustrating the results of transferringthe geometrical details of a tissue to a synthetic rectangle.

FIG. 6 is a series of images showing the results of geometric detailtransferring from a tissue to the image of a piece of wood.

FIG. 7 is a series of images showing the result of transferring thegeometrical details of the same tissue to a table surface.

FIGS. 8A-8F is a series of images showing the aging effect synthesisresults between the faces of a young male (a) and an old male (d).

FIGS. 9A and 9B is a set of images showing the aging process withoutapplying the IBSDT technique. The input image is the same as the oneshown in FIG. 8D.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

In the following description of the preferred embodiments of the presentinvention, reference is made to the accompanying drawings which form apart hereof, and in which is shown by way of illustration specificembodiments in which the invention may be practiced. It is understoodthat other embodiments may be utilized and structural changes may bemade without departing from the scope of the present invention.

1.0 Exemplary Operating Environment

FIG. 1 illustrates an example of a suitable computing system environment100 on which the invention may be implemented. The computing systemenvironment 100 is only one example of a suitable computing environmentand is not intended to suggest any limitation as to the scope of use orfunctionality of the invention. Neither should the computing environment100 be interpreted as having any dependency or requirement relating toany one or combination of components illustrated in the exemplaryoperating environment 100.

The invention is operational with numerous other general purpose orspecial purpose computing system environments or configurations.Examples of well known computing systems, environments, and/orconfigurations that may be suitable for use with the invention include,but are not limited to, personal computers, server computers, hand-heldor laptop devices, multiprocessor systems, microprocessor-based systems,set top boxes, programmable consumer electronics, network PCs,minicomputers, mainframe computers, distributed computing environmentsthat include any of the above systems or devices, and the like.

The invention may be described in the general context ofcomputer-executable instructions, such as program modules, beingexecuted by a computer. Generally, program modules include routines,programs, objects, components, data structures, etc. that performparticular tasks or implement particular abstract data types. Theinvention may also be practiced in distributed computing environmentswhere tasks are performed by remote processing devices that are linkedthrough a communications network. In a distributed computingenvironment, program modules may be located in both local and remotecomputer storage media including memory storage devices.

With reference to FIG. 1, an exemplary system for implementing theinvention includes a general purpose computing device in the form of acomputer 110. Components of computer 110 may include, but are notlimited to, a processing unit 120, a system memory 130, and a system bus121 that couples various system components including the system memoryto the processing unit 120. The system bus 121 may be any of severaltypes of bus structures including a memory bus or memory controller, aperipheral bus, and a local bus using any of a variety of busarchitectures. By way of example, and not limitation, such architecturesinclude Industry Standard Architecture (ISA) bus, Micro ChannelArchitecture (MCA) bus, Enhanced ISA (EISA) bus, Video ElectronicsStandards Association (VESA) local bus, and Peripheral ComponentInterconnect (PCI) bus also known as Mezzanine bus.

Computer 110 typically includes a variety of computer readable media.Computer readable media can be any available media that can be accessedby computer 110 and includes both volatile and nonvolatile media,removable and non-removable media. By way of example, and notlimitation, computer readable media may comprise computer storage mediaand communication media. Computer storage media includes both volatileand nonvolatile, removable and non-removable media implemented in anymethod or technology for storage of information such as computerreadable instructions, data structures, program modules or other data.Computer storage media includes, but is not limited to, RAM, ROM,EEPROM, flash memory or other memory technology, CD-ROM, digitalversatile disks (DVD) or other optical disk storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or any other medium which can be used to store the desired informationand which can be accessed by computer 110. Communication media typicallyembodies computer readable instructions, data structures, programmodules or other data in a modulated data signal such as a carrier waveor other transport mechanism and includes any information deliverymedia. The term “modulated data signal” means a signal that has one ormore of its characteristics set or changed in such a manner as to encodeinformation in the signal. By way of example, and not limitation,communication media includes wired media such as a wired network ordirect-wired connection, and wireless media such as acoustic, RF,infrared and other wireless media. Combinations of the any of the aboveshould also be included within the scope of computer readable media.

The system memory 130 includes computer storage media in the form ofvolatile and/or nonvolatile memory such as read only memory (ROM) 131and random access memory (RAM) 132. A basic input/output system 133(BIOS), containing the basic routines that help to transfer informationbetween elements within computer 110, such as during start-up, istypically stored in ROM 131. RAM 132 typically contains data and/orprogram modules that are immediately accessible to and/or presentlybeing operated on by processing unit 120. By way of example, and notlimitation, FIG. 1 illustrates operating system 134, applicationprograms 135, other program modules 136, and program data 137.

The computer 110 may also include other removable/non-removable,volatile/nonvolatile computer storage media. By way of example only,FIG. 1 illustrates a hard disk drive 141 that reads from or writes tonon-removable, nonvolatile magnetic media, a magnetic disk drive 151that reads from or writes to a removable, nonvolatile magnetic disk 152,and an optical disk drive 155 that reads from or writes to a removable,nonvolatile optical disk 156 such as a CD ROM or other optical media.Other removable/non-removable, volatile/nonvolatile computer storagemedia that can be used in the exemplary operating environment include,but are not limited to, magnetic tape cassettes, flash memory cards,digital versatile disks, digital video tape, solid state RAM, solidstate ROM, and the like. The hard disk drive 141 is typically connectedto the system bus 121 through an non-removable memory interface such asinterface 140, and magnetic disk drive 151 and optical disk drive 155are typically connected to the system bus 121 by a removable memoryinterface, such as interface 150.

The drives and their associated computer storage media discussed aboveand illustrated in FIG. 1, provide storage of computer readableinstructions, data structures, program modules and other data for thecomputer 110. In FIG. 1, for example, hard disk drive 141 is illustratedas storing operating system 144, application programs 145, other programmodules 146, and program data 147. Note that these components can eitherbe the same as or different from operating system 134, applicationprograms 135, other program modules 136, and program data 137. Operatingsystem 144, application programs 145, other program modules 146, andprogram data 147 are given different numbers here to illustrate that, ata minimum, they are different copies. A user may enter commands andinformation into the computer 110 through input devices such as akeyboard 162 and pointing device 161, commonly referred to as a mouse,trackball or touch pad. Other input devices (not shown) may include amicrophone, joystick, game pad, satellite dish, scanner, or the like.These and other input devices are often connected to the processing unit120 through a user input interface 160 that is coupled to the system bus121, but may be connected by other interface and bus structures, such asa parallel port, game port or a universal serial bus (USB). A monitor191 or other type of display device is also connected to the system bus121 via an interface, such as a video interface 190. In addition to themonitor, computers may also include other peripheral output devices suchas speakers 197 and printer 196, which may be connected through anoutput peripheral interface 195. Of particular significance to thepresent invention, a camera 163 (such as a digital/electronic still orvideo camera, or film/photographic scanner) capable of capturing asequence of images 164 can also be included as an input device to thepersonal computer 110. Further, while just one camera is depicted,multiple cameras could be included as an input device to the personalcomputer 110. The images 164 from the one or more cameras are input intothe computer 110 via an appropriate camera interface 165. This interface165 is connected to the system bus 121, thereby allowing the images tobe routed to and stored in the RAM 132, or one of the other data storagedevices associated with the computer 110. However, it is noted thatimage data can be input into the computer 110 from any of theaforementioned computer-readable media as well, without requiring theuse of the camera 163.

The computer 110 may operate in a networked environment using logicalconnections to one or more remote computers, such as a remote computer180. The remote computer 180 may be a personal computer, a server, arouter, a network PC, a peer device or other common network node, andtypically includes many or all of the elements described above relativeto the computer 110, although only a memory storage device 181 has beenillustrated in FIG. 1. The logical connections depicted in FIG. 1include a local area network (LAN) 171 and a wide area network (WAN)173, but may also include other networks. Such networking environmentsare commonplace in offices, enterprise-wide computer networks, intranetsand the Internet.

When used in a LAN networking environment, the computer 110 is connectedto the LAN 171 through a network interface or adapter 170. When used ina WAN networking environment, the computer 110 typically includes amodem 172 or other means for establishing communications over the WAN173, such as the Internet. The modem 172, which may be internal orexternal, may be connected to the system bus 121 via the user inputinterface 160, or other appropriate mechanism. In a networkedenvironment, program modules depicted relative to the computer 110, orportions thereof, may be stored in the remote memory storage device. Byway of example, and not limitation, FIG. 1 illustrates remoteapplication programs 185 as residing on memory device 181. It will beappreciated that the network connections shown are exemplary and othermeans of establishing a communications link between the computers may beused.

The exemplary operating environment having now been discussed, theremaining parts of this description section will be devoted to adescription of the program modules embodying the invention.

2.0 Image-Based Surface Detail Transfer System and Method.

In this section, the system and method of Image-Based Surface DetailTransfer (IBSDT) is described.

2.1 Overview

The surface detail transfer method according to the present invention isimage-based. The idea of changing object appearance with only imageinformation has been explored by various other researchers in bothcomputer vision and graphics communities as discussed in the Backgroundsection.

As discussed in the Background, Burson and Nancy [4] computed thedifference of the aligned images of a young face and an old face. Giventhe image of a new person's face to be aged, the difference image iswarped and added to this new face to make it look older. One problemwith this technique is that the difference image contains the skin colorinformation of the original two faces so that the skin color of the newface may be modified by the aging process (dark skin becomes light skin,etc).

Another method of transferring geometric details from one object toanother was derived by Liu et al. Liu et al. [13] used the image ratiobetween a neutral face and an expression face of the same person (calledexpression ratio image) to modify a different person's neutral faceimage and generate facial expression details.

The IBSDT technique of the invention is related to the works of Liu etal. [13] and Burson and Nancy [4] in that all deal with surfacedeformations. However, the IBSDT method differentiates from these twoworks and all the related works mentioned above in the Backgroundsection in that the IBSDT technique only requires one source image. Thekey observation that is the basis of the IBSDT technique is that thesmoothing in the image domain corresponds to the smoothing in thegeometrical domain when the surface reflectance is smooth. This pointwill be detailed mathematically below.

2.2 Image-Based Surface Detail Transfer Theory

Surface normal is one of the most important geometric factors thatdetermine the visual appearance of a surface. Techniques such as bumpmaps have been used to generate the illusions of modified geometrywithout explicitly changing the shape of the surface. The IBSDT methodoperates by producing the appearance of modifying a surface's normal bytransferring geometric details from another surface without explicit 3Dinformation. For example, given images of two spheres with differentmaterial properties where one sphere is smooth and the other is bumpy,the method can modify the image of the smooth sphere so that it looks asbumpy as the second sphere while its material properties (e.g., color)are kept the same. This can be done without using explicit 3Dinformation as long as the objects are similarly shaped.

The bumps on the surfaces are regarded as the geometrical detailsbecause they represent the high frequency geometrical information. Sincethe method transfers bumps from one surface to another directly fromtheir images without 3D information it is called Image-based SurfaceDetail Transfer.

2.2.1 Notation and Problem Statement

For any point P on a surface S, let n(P) denote its normal. Assume thereare m point light sources. Let l_(i) (P), 1≦i≦m, denote the lightdirection from P to the i-th light source, and I_(i) its intensity.Suppose the surface is diffused, and let ρ(P) be its reflectancecoefficient at P. Under Lambertian model, the recorded intensity ofpoint P in the image I is $\begin{matrix}{{I(p)} = {{\rho(P)}{\sum\limits_{i = 1}^{m}{I_{i}{{n(P)} \cdot {l_{i}(P)}}}}}} & (1)\end{matrix}$where p=C(P) is the 2D projection of P onto the image, and C (·) is thecamera projection function. Two surfaces S₁ and S₂ are said to bealigned if there exists a one-to-one mapping F such that for all P₁εS₁and P₂=F(P₁)εS₂∥P ₁ −P ₂∥≦ε  (2)where ε is a small positive, and furthermore, there exist neighborhoodsΘ(P₁) of P₁ and Θ(P₂) of P₂ such that∥{overscore (n)}P ₁ −{overscore (n)}P ₂∥≦δ  (3)where δ is a small positive, and {overscore (n)}(P₁) and {overscore(n)}(P₂) are the mean normal defined in the neighborhoods of Θ(P1) of P1and Θ(P₂) of P₂, respectively. The problem can then be stated as thefollowing. Given images I₁ and I₂ of two aligned surfaces S₁ and S₂,respectively, what is the new image I₂′ of S₂ if its surface normal ismodified such thatn ₂′(P ₂)=n ₁(P ₁)  (4)where P₁ and P₂ are the corresponding points defined by the mapping F.

2.2.2 A Geometric Viewpoint

The following discussion assumes a single point light source to simplifythe derivation. Extension to multiple light sources is straight forward.Because the distance between P₁ and P₂ is small according to Eq. (2), itis reasonable to assume that the light is always sitting far away enoughsuch that ε<<d_(l), where d_(l) is the average distance from light tothe points. This leads to the approximation l (P₁)≈l(P₂). From Eq. (1)and (4), it can be shown that $\begin{matrix}\begin{matrix}{\frac{I_{2}^{\prime}\left( p_{2} \right)}{I_{2}\left( p_{2} \right)} \equiv \frac{\rho\quad\left( P_{2} \right)l\quad{{n_{2}\left( P_{2} \right)} \cdot {l\left( P_{2} \right)}}}{{\rho\left( P_{2} \right)}l\quad{{n_{2}\left( P_{2} \right)} \cdot l}\quad\left( P_{2} \right)}} \\{\frac{I_{2}^{\prime}\left( p_{2} \right)}{I_{2}\left( p_{2} \right)} \approx {\frac{{\rho\left( P_{1} \right)}l\quad{{n_{1}\left( P_{1} \right)} \cdot {l\left( P_{1} \right)}}}{{\rho\left( P_{2} \right)}l\quad{{n_{2}\left( P_{2} \right)} \cdot {l\left( P_{2} \right)}}}\frac{\rho\left( P_{2} \right)}{\rho\left( P_{1} \right)}}} \\{\equiv \frac{{I_{1}\left( p_{1} \right)}{\rho\left( P_{2} \right)}}{{I_{2}\left( p_{2} \right)}{\rho\left( P_{1} \right)}}}\end{matrix} & (5)\end{matrix}$where ρ has the same meaning as in the Eq. (1), p₁=C₁(P₁), p₂=C₂(P₂),and I₁, I₂, and I′₂ have the same meaning as in the problem statement.Notice that the C (·) functions are different for the two surfaces. Thisis because the images I₁ and I₂ of the surfaces could be taken by twodifferent cameras. This leads to $\begin{matrix}{{I_{2}^{\prime}\left( p_{2} \right)} \approx \frac{{I_{1}\left( p_{1} \right)}{\rho\left( P_{2} \right)}}{\rho\left( P_{1} \right)}} & (6)\end{matrix}$In order to compute the ratio of ρ(P₁) and ρ(P₂), the smoothed image ofI is defined as $\begin{matrix}{{\overset{\_}{I}(p)} \equiv {\sum\limits_{q \in {\Omega{(p)}}}{{w(q)}{I(q)}}}} & (7)\end{matrix}$where ΩΩ(p)=C(Θ(P)) is the neighborhood of p, and w is the kernelfunction of a smooth filter, say, a Gaussian filter or an averagefilter. Assuming that the size of Θ(P) is relatively small as comparedwith its distance to the light source, l(P)≈l(Q), ∀ QεΘ(P). Alsoassuming that ρ(P)≈ρ(Q)∀ QεΘ(P), from Eq. (7) and Eq. (1), it is thenobvious that $\begin{matrix}{{\overset{\_}{I}(p)} \equiv {{\rho(P)}{{l\left( {\sum\limits_{Q \in \Theta}{{w\left( {C(Q)} \right)}{n(Q)}}} \right)} \cdot {l(P)}}}} & (8)\end{matrix}$where${{\sum\limits_{Q \in \Theta}{{w\left( {C(Q)} \right)}{n(Q)}}} = {\overset{\_}{n}(P)}},$and {overscore (n)}(P) is the mean normal as mentioned in the problemstatement. For surface S₁ and S₂, then $\begin{matrix}{\frac{{\overset{\_}{I}}_{2}\left( p_{2} \right)}{{\overset{\_}{I}}_{1}\left( p_{1} \right)} \equiv \frac{{\rho\left( P_{2} \right)}l\quad{{\overset{\_}{n}\left( p_{2} \right)} \cdot 1}\left( P_{2} \right)}{{\rho\left( P_{1} \right)}l\quad{{\overset{\_}{n}\left( P_{1} \right)} \cdot 1}\left( P_{1} \right)}} & (9)\end{matrix}$Since the two surfaces are aligned, l(P₁)≈l(P₂), and {overscore(n)}(P₂)≈{overscore (n)}(P₁). Equation (9) can then be rewritten as$\begin{matrix}{\frac{\rho\left( P_{2} \right)}{\rho\left( P_{1} \right)} \approx \frac{{\overset{\_}{I}}_{2}(p)}{{\overset{\_}{I}}_{1}(p)}} & (10)\end{matrix}$Substituting Eq. (10) into Eq. (6) leads to $\begin{matrix}{{I_{2}^{\prime}\left( p_{2} \right)} \approx {\frac{I_{1}\left( p_{1} \right)}{{\overset{\_}{I}}_{1}\left( p_{1} \right)}{{\overset{\_}{I}}_{2}\left( p_{2} \right)}}} & (11)\end{matrix}$Eq. (11) shows that the transfer of surface normal can be approximatedby some simple operations on the images of the surfaces.

2.2.3 An Intuitive Signal Processing Viewpoint

Eq. (11) is rewritten as $\begin{matrix}{{I_{2}^{\prime}\left( p_{2} \right)} \approx {\frac{I_{1}\left( p_{1} \right)}{{\overset{\_}{I}}_{1}\left( p_{1} \right)}{{\overset{\_}{I}}_{2}\left( p_{2} \right)}} \equiv {\left( {1 + \frac{{I_{1}\left( p_{1} \right)} - {{\overset{\_}{I}}_{1}\left( p_{1} \right)}}{{\overset{\_}{I}}_{1}\left( p_{1} \right)}} \right){{\overset{\_}{I}}_{2}\left( p_{2} \right)}}} & (12)\end{matrix}$From a signal processing view point, Eq. (12) simply substitutes thehigh frequency components of I₂ with those from I₁. The high frequencycomponents I₁-Ī₁ in I₁ are normalized by Ī₁ in order to cancel theintensity scale difference between the low frequency components of I₂ inI₁. Generally, I₁ could be any image, regardless of the conditions givenin the previous section. But the resultant image could be meaninglessbecause of the inconsistency between the transferred detailed componentsfrom I₁ and native low frequency components on the I₂. This happens whenI₁ and I₂ are the images of two surfaces that are not aligned.3.0 Implementation of IBSDT

In general, as shown in FIGS. 2A and 2B, given images I₁ and I₂ ofsimilar shapes (process action 202), to perform surface detail transfer,the two images first need to be aligned, as shown in process action 204.For simple geometrical shapes such as rectangles and spheres, usually itis only necessary to perform global transformations including rotation,translation, and scaling (process actions 204 a and 204 b). For morecomplicated shapes such as human faces, markers are first manuallyplaced on the boundaries and the feature points, and then pixelalignment is obtained through image warping [1, 12] (process actions 204c and 204 d). In one implementation of the invention, a simpletriangulation-based image warping method is used. Once the alignment isdone, a Gaussian filter is run with a user specified σ on I₁ and I₂ toobtain smoothed image intensities Ī₁ and Ī₂ of the warped image 1 andimage 2 (process action 206). For each pixel in the first warped images,the ratio of the original intensity to the smoothed intensity iscalculated (process action 210). Finally Equation (11) is used to obtainI₂′ by multiplying the obtained ratio of the original intensity to thesmoothed intensity for every corresponding pixel in I₂ (process action210).

Intuitively, the standard deviation sigma (σ) of the Gaussian filtercontrols how much geometrical smoothing is performed on the surface ofI₁. It determines the scale of the surface details to be transferred. Asmall σ allows fine geometrical details to be transferred while a largeσ allows only large scale geometrical deformations to be transferred.

3.1 Implementation with Respect to A Facial Aging Method

One facial aging embodiment of the IBSDT system and method will now bedescribed.

Geometrically, the difference between an old person's skin surface and ayoung person's skin surface is that the old person's skin surface hasmore bumps than the young face. If the bumps of an old person's skinsurface are transferred to a young person's face, the young person'sface will become bumpy and look older. Conversely, it is also possibleto replace the bumps of an old person's skin surface with that of theyoung person's face so that the old person's face gets smoother andlooks younger. The surface details transfer system and method asdescribed above can be applied to human faces to simulate aging effects.

Since human faces are generally alike, e.g., they all have eyes, nosesand mouths and their shapes are similar, it is relatively easy to alignthe shapes of two faces by simple warping between the images of theface. The aging simulation based on IBSDT is outlined below.

-   -   1. Input. Images of two faces with different ages, taken under        the same or similar lighting conditions are input. Suppose that        I₂ is the image of the face to be aged, and I₁ is the one from        which the surface details will be transferred.    -   2. Image Warping. The feature point set for both images are        marked manually, and I₁ is warped to I₂ according to the image        correspondences. The warped image is denoted as Ĩ₁.    -   3. Image Smoothing. Compute smoothed versions, i.e. Ĩ₁and Ī₂ of        Ĩ₁ and I₂ with Gaussian filter of a specified σ.    -   4. Transferring of Surface Details. Transfer surface details for        Ĩ₁to I₂ using Eq. (11), using the warped version of the original        first image (Ĩ₁) to the smoothed warped version of Ĩ₁in        calculating the ratio that is multiplied by the smoothed Ī₂.    -   5. Output. Output an aged face image I₂′.

The best way to prepare input images is to take the two face imagesunder the same lighting conditions. This will ensure the validity of Eq.(11), and maximize the realism of the aged face image. When the lightingconditions are not radically different, Eq. (11) is still valid totransfer the details of the intensity changes from I₂ to Ĩ₁. Theresultant image contains surface detail information from one lightingcondition and the global surface information from the other. Inpractice, the human eye is much less sensitive to this sort oflocal-global inconsistency than might be expected. Consequently, theaging results usually look reasonable even when the lighting conditionsare quite different.

For images taken under completely different lighting conditions, arelighting technique may be used to compensate for the lightingdifference between the images. For instance, Marschner et al. [14,15]used the color ratio between the rendered image pairs under the old andnew lighting conditions to modify photographs taken under the oldlighting conditions to generate photographs under the new lightingcondition. Similarly, Debevec [6,7] used the color difference (insteadof ratio) between image pairs to modify the original photograph.Riklin-Raviv and Shashua [16] used color ratio (called quotient image)to adjust images taken under different lighting conditions.

Another issue with respect to the IBSDT technique is that it assumesthat the surface reflectances are smooth. For objects with abruptreflectance changes such as small color spots, the IBSDT may confusethese color spots with geometrical details. For example, geometricaldetails such as wrinkles and muscle tone changes are the only changessought to be transferred. However, details such as beauty spots orfrontal hairs extending into the forehead in the face images will alsobe transferred. The reason is that abrupt changes in the reflectance ofthese areas violate the assumption of Eq. 8 that reflectances in aregion should be similar. As a result, the intensity changes areregarded as the normal consequence of normal changes and are transferredas well by the IBSDT. Another limitation is that the IBSDT techniqueonly applies to skin, not eyebrows, eyeballs, or lips. In an embodimentof the invention dealing with facial changes, these regions are maskedout so that they are not changed. Alternately, it may be possible toseparate these color variations from geometry variations perhaps throughlearning or some other approaches.

Image warping is a simple matter of transforming one spatialconfiguration of an image into another. To warp the images, points aremanually marked on the face features, by allowing a person to mark thepoints on the displayed image with an input device. These face featurestypically include points around the circumference of the face, andpoints around the eyes, mouth and nose. Texture mapping software andhardware is used to warp an image from one set of markers to another. Inone embodiment of invention, Delaunay triangulation is simply applied tothe mark points. Delaunay triangulation is well known in image warping.It connects an irregular point set (in this case determined by themarkers on the face features) by a mesh of triangle's each satisfyingthe Delaunay property. This means that no triangle has any points insideits circumcircle, which is the unique circle that contains all threepoints (vertices) of the triangle The warp is realized by applying thetriangular mesh of the first point set in the first image to the secondpoint set in the second image. Each point on each triangle can beuniquely mapped to the corresponding triangle of the second point set byan affine transformation, which basically consist of scaling,translation and skewing. This method is fast but the resulting imagequality is not as good as with other more advanced image warpingtechniques. Other more advanced image warping techniques known in theart can also be used to warp the images.

The standard deviation sigma (σ) of the Gaussian filter is the onlycontrol parameter and plays an important role in the whole procedure. Ina Gaussian filter, the value of the pixel under investigation isreplaced by the Gaussian-weighted average of the pixel values in thefilter region which lie in the interval plus or minus σ from the valueof the pixel that is filtered. As a result, the σ determines the scaleof the surface details to be transferred from Ĩ₁ to I₂. If σ is set tobe small, only fine details on the face such as wrinkles will betransferred. On the contrary, larger scale details such as those causedby the muscle shrink can also be transferred which can also be used.

4.0 Results

FIG. 3 shows the results of transferring the geometrical details of asynthetic sphere to a nectarine. The bumps on the synthetic sphere aregenerated by using bump mapping technique. The surface reflectanceproperty on the synthesized sphere is set to be uniform. A point lightsource is placed on top of the sphere so that its lighting condition issomewhat close to the lighting condition of the nectarine. It can beseen that that the bumps on the synthetic sphere are transferred nicelyto the nectarine except at the bottom where the synthetic sphere isbasically dark. The sizes of the image are 614 by 614 pixels, and σ is8.

FIG. 4 shows the results of transferring the geometrical details of areal orange to the same nectarine as in FIG. 3. The bumps on the orangesare transferred faithfully to the nectarine. The image dimensions and aare the same as in FIG. 3. This example also reveals a limitation of theIBSDT procedure: the highlights on the orange are transferred to thenectarine. The reason is that the highlights are treated as being causedby geometrical variations.

FIG. 5 shows the results of transferring the geometrical details of atissue to a synthetic rectangle. It can be seen that only thegeometrical bumps on the tissues are transferred to the rectangle whilethe material color of the rectangle is preserved.

FIG. 6 shows the results of geometric detail transferring from the sametissue to the image of a piece of wood. Both pictures are taken underthe same lighting conditions. It can be seen that the small bumps on thetissues are transferred to the wood while the wood texture is preserved.

FIG. 7 shows the result of transferring the geometrical details of thesame tissue to a table surface. This table surface has a differenttexture pattern than the wood in FIG. 6. It is interesting to comparethe results (the images on the right) in FIG. 6 with FIG. 7, and noticethat they have the same geometrical bumps but different materialproperties.

One interesting application of IBSDT is aging effect synthesis.Geometrically, the difference between an old person's skin surface and ayoung person's skin surface is that the old person's skin surface hasmore bumps than the young face. If the bumps of an old person's skinsurface are transferred to a young person's face, the young person'sface will become bumpy and look older. Conversely, the bumps of an oldperson's skin surface can also be replaced with that of the youngperson's face so that the old person's face gets smoother and lookyounger. The IBSDT technique as described above is shown as it isapplied to human faces to generate aging effects. The alignment is doneby first marking face boundaries and face features such as eyes, noses,and mouths, and then use triangulation-based image warping to warp I₁toward I₂. In one embodiment IBSDT is only applied to pixels inside ofthe face boundary. In addition, the pixels in the regions of the twobrows, the two eyeballs, nose top, and the mouth are not modified byIBSDT either.

FIGS. 8A-8F show the aging effect synthesis results between the faces ofa young male FIG. 8A and an old male FIG. 8D. For each face, a differenta of the Gaussian filter is used during the surface detail transfer.Images in the middle (FIG. 8B and FIG. 8E) are the results with σ=3, andthose on the right (FIG. 8C and FIG. 8F) with σ=8. It can be seen thatvarying σ produces reasonable in-between aging effects such as FIG. 8Band FIG. 8E. Obviously, surface detail transfer plays an important rolewhen making a young person older. However, it is less apparent why thistechnique is necessary to make an old person younger. To clarify thispoint, FIG. 8D is simply smoothed without transferring surface detailsfrom FIG. 8A, while masking out the facial features as before. FIG. 9Ashows the results with σ=3 and σ=8 is shown in FIG. 9B. As compared withthe images in FIGS. 8E and 8F with the same σ's, it can be seen thatimages in FIGS. 9A and 9B are much less sharp and convincing.

The foregoing description of the invention has been presented for thepurposes of illustration and description. It is not intended to beexhaustive or to limit the invention to the precise form disclosed. Manymodifications and variations are possible in light of the aboveteaching. It is intended that the scope of the invention be limited notby this detailed description, but rather by the claims appended hereto.

REFERENCES

-   [1] T. Beier and S. Neely. Feature-based image metamorphosis. In    Computer Graphics, pages 35-42. Siggraph, July 1992.-   [2] Boissieux, G. Kiss, N. m. Thalman and P. Karla. Simulation of    Skin Aging and Wrinkles with Cosmetic Insights. In Computer    Animation and Simulation 2000, 2000.-   [3] J. Blinn, Models of light reflection for computer synthesized    pictures. In Computer Graphics, pages 192-198, SIG-GRAPH, July 1977.-   [4] N. Burson and T. D. Schneider. Method and apparatus for    producing an image of a person's face at a different age. U.S. Pat.    No. 4,276,570, 1981.-   [5] C. Choi, Age Change for Predicting Future Faces. In IEEE    International Fuzzy Systems Conference Proceedings, pages 1603-1608,    1999.-   [6] P. E. Debevec. Rendering synthetic objects into real scenes:    Bridging traditional and image-based graphics with global    illumination and high dynamic range photography. In Computer    Graphics, Annual Conference Series, pages 189-198.-   [7] P. E. Debevec, T. Hawkins, C. Tchou, H.-P. Duiker, W. Sarokin,    and M. Sagar. Acquiring the reflectance field of a human face. In    Computer Graphics, Annual Conference Series, pages 145-156.    Siggraph, July 2000.-   [8] R. Epstein, A. Yuille, and P. Belhumeur. Learning object    representations from lighting variations. In ECCV 96 International    Workshop, pages 179-199, 1996.-   [9] G. Healey and T. Binford. Local shape from specularity. Computer    Vision Graphics and Image Processing, pages 62-86, April 1988.-   [10] B. Horn and M. J. Brooks. Shape from Shading. MIT Press, 1989.-   [11] A. Lanitis, C. Taylor, and T. Cootes. Modeling the process of    aging in face images. In IEEE Proceedings of the 7th International    Conference on Computer Vision, pages 131-136.-   [12] P. Litwinowicz and L. Williams. Animating images with drawings.    In Computer Graphics, pages 235-242. Siggraph, August 1990.-   [13] Z. Liu, Y. Shan, and Z. Zhang. Expressive expression mapping    with ratio images. In Computer Graphics, Annual Conference Series,    pages 271-276. Siggraph, August 2001.-   [14] S. R. Marschner and D. P. Greenberg. Inverse lighting for    photography. In IST/SID Fifth Color Imaging Conference, November    1997.-   [15] S. R. Marschner, B. Guenter, and S. Raghupathy. Modeling and    rendering for realistic facial animation. In Rendering Techniques,    pages 231-242. Springer Wien New York, 2000.-   [16] T. Riklin-Raviv and A. Shashua. The quotient image: Class based    rerendering and recongnition with varying illuminations. In IEEE    Conference on Computer Vision and Pattern Recognition, pages    566-571, June 1999.-   [17] H. Rushmeier, G. Taubin, and A. Gueziec. Applying shape from    lighting variation to bump map capture. In Eurographics Workshop on    Rendering, pages 35-44, 1997.-   [18] Y. Wu, P. Kalra, and N. M. Thalmann. Physically-based wrinkle    simulation & skin rendering. In Erographics Workshop on Computer    Animation and Simulation, pages 69-79, 1997.

1. A computer-implemented process for transferring geometrical detailsfrom a first object in one image to a second object in a second image,comprising using a computer to perform the following process actions:inputting a first image depicting an object whose surface details are tobe transferred and a second image depicting an object of similar shapeas depicted in the first image; aligning the objects of similar shape infirst and second images; computing a smoothed version of the first andsecond images; for each pixel in the first aligned image, computing theratio of the original intensity to the smoothed intensity; creating anew image by multiplying the ratio of the original intensity to thesmoothed intensity of the first image by the smoothed intensity for eachcorresponding point in the second image.
 2. The computer implementedprocess of claim 1 wherein the process action of aligning the objects ofsimilar shape in the first and second images comprises the processactions of: performing a rotation of the images to rotationally alignthe object in the second image with the object of similar shape in thefirst image; performing a translation of the images to align the objectin second image with the object of similar shape in the first image; andperforming a scaling of the images to match the size of the object inthe second image with the object of similar shape in first image.
 3. Thecomputer-implemented process of claim 1 wherein the process action ofaligning the objects of similar shape in the first and second imagescomprises the process actions of: establishing a standard map havingnodes at the point of important features; overlaying said map on thefirst image, so its nodes match the important features; overlaying saidmap on the second image, and causing it to match the important featuresthereof; and using coordinates of said map on said second image to warpsaid second image so as to match the first image, thereby allowing thecorresponding pixel locations in the first and second images as well astheir respective intensities to be identified.
 4. Thecomputer-implemented process of claim 1 wherein the process action ofcomputing a smoothed version of the first and second images comprisesperforming a Gaussian filtering procedure to each pixel in the first andsecond images to compute a new value of intensity for each pixel.
 5. Thecomputer-implemented process of claim 4 wherein the standard deviationof the Gaussian filter determines the level of the surface details to betransferred from the first image to the newly created image.
 6. Thecomputer-implemented process of claim 5, wherein the standard deviationof the Gaussian filter is set to be small and only fine details aretransferred.
 7. The computer-implemented process of claim 5, wherein thestandard deviation of the Gaussian filter is set to be larger and largerscale details are transferred.
 8. The computer-implemented process ofclaim 1 wherein the first and second images do not exhibit similarlighting conditions and wherein a relighting technique is applied tocompensate in the lighting conditions between the first and secondimage.
 9. A system for simulating the appearance of a face at adifferent age, the system comprising: a general purpose computingdevice; and a computer program comprising program modules executable bythe computing device, wherein the computing device is directed by theprogram modules of the computer program to, input a first imagedepicting a face whose facial characteristics are to be transferred anda second image depicting a face that is to receive the facialcharacteristics of the face in the first face; align the faces in firstand second images; compute a smoothed version of the faces in the firstand second images; for each pixel in the face of the first alignedimage, compute the ratio of the original intensity to the smoothedintensity; and create a new image by multiplying the ratio of theoriginal intensity to the smoothed intensity of the face in the firstimage by the smoothed intensity for each corresponding point in thesecond image.
 10. The system of claim 9 wherein the computer module toalign the faces comprises sub-modules to: establish a standard map ofmarkers at the point of important facial features; overlay said standardmap on the face in the first image, so its markers match the importantfacial features; overlay said standard map on the face in the secondimage, and causing it to match the important features thereof; and usecoordinates of said map on said face in the second image to warp saidsecond image so as to match the first image, thereby allowing thecorresponding pixel locations in the first and second images as well astheir respective intensities to be identified.
 11. The system of claim 9wherein the facial characteristics of the face in the first image depictan older face than the facial characteristics of the face in the secondimage, and wherein the newly created image depicts an older face thanthe face depicted in the second image.
 12. The system of claim 9 whereinthe facial characteristics of the face in the first image depict ayounger face than the facial characteristics of the face in the secondimage, and wherein the newly created image depicts a younger face thanthe face depicted in the second image.
 13. The system of claim 9 whereinthe module for smoothing the first and second image comprisessub-modules to: for a given area of pixels of predefined size, averagethe pixel intensity by averaging the intensity of the surroundingpixels.
 14. The system of claim 9 wherein the process action ofcomputing a smoothed version of the first and second images comprises:applying a Gaussian filter to each pixel in the first and second images.15. The system of claim 9 wherein certain areas of the first and secondimages are masked so that they are not changed in the created new image.16. The system of claim 9 wherein the first image depicting a face whosefacial characteristics are to be transferred is a face of different age,race or gender from the face in the second image.
 17. Acomputer-readable medium having computer-executable instructions fortransferring the surface details of an object in one image to an objectof similar shape in another image, said computer executable instructionscomprising: recording a first image depicting an object whose surfacedetails are to be transferred and a second image depicting an object ofsimilar shape as depicted in the first image: smoothing first and secondimages; for each pixel in the first image, computing the ratio of theoriginal intensity to the smoothed intensity; and creating a new imageby multiplying the ratio of the original intensity to the smoothedintensity of the first image by the smoothed intensity for eachcorresponding point in the second image.
 18. The computer-readablemedium of claim 17 wherein the object in the first image is a real-worldobject and the object in the second image is a synthetic object.
 19. Thecomputer-readable medium of claim 17 further comprising computerexecutable instructions for warping the images prior to smoothing themso that the objects of similar shape a located in the same location inthe first and second images.